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ABSTRACT 

An algorithm is proposed for obtaining local surface orientation from the apparent distortion of 
surface patterns in an image. 

A spherical projection is used for imaging. A mapping is defined from points on this image sphere 
to a locus of points on the gaussian sphere which corresponds to possible surface orientations. This 
mapping is based on the measurement of the local distortions of a repeated known texture pattern 
due to the imaging projection. This locus of possible surface orientations can be reduced to a unique 
orientation at each point on the image sphere using 3 vantage points and taking the intersection of 
the loci of possible orientations derived from each vantage. It is also possible to derive a unique surface 
orientation at each image point through the use of an iterative constrain', propagation technique along 
with the orientation information available at occluding boundaries. Both method are demonstrated for 
real images. 
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0 OVERVIEW OF THIS PAPER 
0.1 Main Result 




Fig. 1 A picture of a golf ball. 


The distortion of these circles on the golf ball can be 
used to recover local surface orientations. 


Fig. 2 A obtained surface. 
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0.2 Summary of the Approach 1 

(l)We use a spherical projection to treat the perspective view. 

(A NEW TOOL TO DEVELOP A CONSTRAINT) 

We will propose to use a spherical surface as an image surface. This gives us an image surface perpen¬ 
dicular to all lines of sight from the viewer. 


(2) We propose a mapping which expresses regular-pattern gradients. 

(A STATIC CONSTRAINT TO BE USED IN THE PROPAGATION TECHNIQUE) 

Although Render proposed two kinds of maps for texture gradients, neither of them can be applied 
to our shape-from-regular-pattern problem. The first one docs not have the ability to express surface 
orientations from regular patterns on a surface. The second one requires aggregation of many lines to 


determine any one direction. Thus, we consider here the 


way in which the distortion of a regular pattern 


depends on surface orientations. 


(3) Our iterative propagation method for the shape-from-shading problem can also 
be used to solve the shape-from-regular-pattern problem. 

(HOW TO PROPAGATE CONSTRAINTS) 

We have proposed an iterative propagation method for the shape-from-shading problem. If we can 
describe regular-pattern gradients in the same way that the reflectance map expresses shading informa¬ 
tion, we can apply the algorithm to the shape-from-regular-pattern problem. Moreover, the shape-from- 
regular-pattcrn-problem always involves observable occluding boundaries required by the algorithm. 

(4) We also consider a technique for determining the surface orientation of a planer 
surface without iteration. 

(ENOUGH STATIC CONSTRAINTS CAN ALSO SOLVE A PROBLEM) 

If a surface contains more than three regular patterns and if we can assume that the surface is planar, we 
can determine its orientation without iteration using a method similar to the photo metric stereo method. 


1 Note that we will use two spheres. One sphere is the image surface. The other sphere (a Gaussian sphere) is used to express 
surface orientations. Both of the spheres are projected onto a plane using the stcrcographic projection for illustration in this paper. 
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1 A SPHERICAL PROJECTION 


We propose a spherical projection to treat the perspective view. An observer is located at the center of the 
sphere. Ail image contours are projected on this sphere with respect to its center. We will use the term image 
sphere as opposed to image plane in this paper to refer to the spherical projection. Every point in the real world 
is projected onto die image sphere with respect to its center. All lines of sight of the viewer are extended radii 
of die sphere and thus arc perpendicular to the image plane. 




Fig. 3 An Image Sphere. 

All contours are projected on the image sphere with respect to the center. 




All contours on the image sphere are foreshortened by the relationship between the direction of the surface 


normal and the line of sight. At each point of the image sphere the line of sight is perpendicular to the image 


sphere, so there is no foreshortening of the image due to the position of a pattern on an image. 2 Thus, all 
contours on the image sphere arc due to the relationship between a line of sight and the surface normal of the 


2 t 


Fhe perspective projection is different from the spherical projection. In the perspective projection a pattern on an image is 
distorted twice. The first distortion is due to the relationship between the viewed surface normal and the direction of viewer’s line 
of sight. The second one is due to the relationship between the line of sight and the image plane. Thus, the second distortion 
depends on the position of a pattern on the image plane. 
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surface patch viewed from tire viewer. 


The essential point of a perspective view obtained with the spherical projection there exists only one viewing 
position, the viewer observes and interprets the outer world with respect to his position alone. He does not 
have to compensate for die orientation of an image plane. For example, consider the perspective projection 
of an extended planer surface parallel to die image plane which is covered with small squares (Fig. 4). 
The squares on die plane always project as squares, no matter where the squares lie on the surface. With 
die spherical projection die squares would appear as various rectangles depending on their positions on the 
surface. 



Fig. 4 A Defect of the Perspective Projection. 

The so-called perspective projection cannot express foreshortened cficct when an object exists on the plane 
parallel to the image plane. 


The spherical projection has several advantages over the conventional perspective projection. 


(1) The local geometry in an image docs not change when the orientation of die viewer changes. This is 
not die case with the conventional perspective projection where a change in the orientation of the image 
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plane can cause a dramatic change in the resulting image geometry. Thus, for the spherical projection 
distortion of a pattern depends only on the direction of line of sight and the direction of the surface 
normal. 


(2) The spherical projection can treat 360 degree scenes. 


(3) We can use the same regular-pattern gradient map at each image point in the spherical projection, 
because each contour depends only on the relationship between die direction of line of sight and the 
direction of die viewed surface normal. 


The spherical projection also has psychological validity. The most sensitive area of the retina (the forvea) has 
an extent of only 3-4 degrees over the visual field. Eve movements (and head movements) are required to 
examine patterns in different directions. Thus, the line of sight is always perpendicular to die image plane. It 
is interesting that some kinds of dragonfly have compound eyes which give rise to projection like that in the 


image sphere. Most photographs taken by cameras cover no more dian a 20 degree field of view. Widiin this 
range, one can regard the image plane as a part of a sphere. 3 


3 

' Appendix 2 shows how to convert an image on the perspective projection into an image on the spherical projection. However, 
this adjustment is usually unnecessary, because the extra distortion of the perspective projection is proportional to the cosine of 
the angle between the direction of the optical axis and the direction of the line of sight, namely cos 20 deg or 0.94. Thus, the 
distortion due to the perspective projection is small compare with the distortion due to the oiicntation of the viewed surface. 
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2 MEASURING TEXTURE DISTORTION DUE TO SURFACE ORIENTATION 

2.1 Definition of the Problem 

We will determine surface orientation based on the apparent distortion of regular patterns on an image sphere, 
provided that: 

(0 The surface is covered with a uniform texture of repeated texture element. 

We call die uniform texture as a regular pattern. 

(2) Each texture element is small, compared with the distance between the viewer and the viewed surface. 

If the texture element is small, the texture element can be regarded as projected onto a tangential plane 
of the image sphere. All apparent distortion of the texture element is caused between the direction of the 
line of sight from the viewer and die direction of the surface normal. 

(3) Each texture element is small, compared with a change of surface orientation there. 

Each texture element is assumed to lie in die plane of the viewed surface. 


(4) The original shape of the texture element (a generator) is known. 

Since we will decode surface orientations from the apparent distortion of a pattern, it is necessary for us 
to have a standard pattern with which to compare die distorted pattern. This requirement is similar to 
that of the albedo ratio in using the reflectance map. If one docs not know the albedo ratio, one cannot 
make a correspondence between actual intensities and brightness values on the reflectance map. 


2.2 A Measure for Distortion in Regular Patterns 

In this section we examine the relationship between surface orientation and the distortion of regular patterns. 
Fig. 5 shows the projections of squares on a plane inclined 45 degrees in die vertical direction. Even diough 
the generators (squares) lie on the same tilted plane, their images arc different from each other. Their distor¬ 
tion depend on two factors. One factor is the orientation of the surface. This is what we arc interested in. 
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The other is the orientation of the squares in the plane of the surface. This in surface rotation causes the 
variations shown in Fig. 5. Our goal is to find some intrinsic measurement that depends only on surface 
orientation. Render's first method[l] treats only a few particular cases: when edges of the pattern are parallel 
to the coordinate axis and the edges are perpendicular to each other (only case A or case D in Fig. 5). Because 
the first method only takes a ratio of apparent lengths of two edges. The map cannot be applied to case B nor 


case C in Fig. 5. Render's second mcthod[2] requires a large number 


of edge vectors in order to determine 


a single surface orientation. This method fails, when a surface has different orientations everywhere. For 


example, Render's second method will not decode surface orientation from the distortion of the small circles 


on the golf ball in Fig. 


1. What we want is a way of measuring surface orientation independent of the in 


surface orientation of the texture elements. 





Fig. 5 Projections of Squares ^ . , 

All squares lie on a plane inclined in north-south 45 degrees and east-west 0 degrees. Their distoition depend 

on both the orientation of the surface and the orientation of the squares in the plane of the surface. 
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If two perpendicular axis vectors can be associated with a texture element in a unique manner, we can use this 
pair of vectors as a kind of generalized texture element (See Fig. 6). When a pattern is distorted because of 
the angle between the direction of die surface normal and the direction of the viewer, the two vectors will also 


be distorted. This distortion of two axis vectors is die same as the distortion of the regular pattern, because the 


axis vectors are fixed on the regular pattern. We will, therefore, 


concentrate on the distortion of these two axis 


vectors. 



Fig. 6 

Pairs of axis vectors 


The calculation in Appendix l shows us that the magnitude of the cross 


product of two axis vector projections 
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is proportional to cos a j and tire sum of squares of their lengths is proportional to 1 -j- cos 2 w where w is 


the angle between the direction of the viewer and the direction of the surface orientation. These two values 
depend only on the surface orientation; and they are independent on the rotation angle of the regular pattern. 
For example, the patterns in Fig. 5 will yield tire same value since they are all on tire same tilted surface. 

The ratio of the cross product of the two axis vectors to the sum of the lengths of the two axis vectors is 
the desired measure for the regular-pattern gradient map. The length of a small object on the image sphere 
depends on the distance between the viewer and the object. A distant object is small while near objects are 
large. Both the cross product and the sum of lengths depend on the distance, but these effects cancel when 
their ratio is taken as shown in Appendix 1. Thus, the ratio is independent of the distance between the viewer 
and the pattern. This ratio is also independent of the viewer’s orientation, because of the spherical projection. 
The ratio only depends on the angle between the direction of the viewer and the direction of the surface 
orientation. This means that we can use the same regular-pattern gradient map everywhere on the image 
sphere. 


We will call I in Eq. I the distortion value. 


I 


fg sin r 
f 2 + <J 2 ’ 


( 1 ) 


where / and g are the observed lengths of axis vectors on the image sphere and r is the angle between the two 
projected axis vectors. /, g, and r can be measured directly from an image.' 1 Thus, / can be obtained from 
a image, no matter how the pattern is rotated or reduced due to distance. I has the following relation with 


surface orientation: 


I 


COSO) 


1 -f- COS 2 U), 


( 2 ) 


where to is the angle between the direction of the viewer’s line of sight and the direction of the surface normal. 

This / will be the basis of the regular-pattern gradient map in the discussion to follow. 

Ihc two axis vectors have different three-dimensional lengths, this difference can be corrected for at the time of measurement 
since the projected length of each vector is proportional to its actual length. Namely g' ~ fg where g is the measured length of 
the second axis vector and t is the ratio of the three-dimensional length of the first axis vector over the length of the second. 
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2.3 The Gaussian Sphere (A Method for Representing Surface Orientations) 

Surface orientations can be associated with points on the gaussian sphere. 5 The gaussian sphere is a sphere of 
unit radius whose Z-axis is taken as an extended line through the north and south poles of the sphere. Assume 
that we put a surface patch of an object at the center of the sphere and that the direction of the viewer is the 
direction from the center to the north pole. The surface patch faces some point on the sphere. For example, 
if a surface normal has die same direction as the south pole, the surface patch is perpendicular to the viewer’s 
line of sight. All possible orientations of s surface patch are represented on the gaussian sphere. Thus, we can 
describe local surface orientation on an object in terms of points on the gaussian sphere. 

2.4 A Regular-Pattern-Gradient Map 

We can associate the distortion value I with points on the gaussian sphere. Since die distortion value I 
depends only on the angle w between the direction of the surf ce normal and die direction of viewer’s line of 

sight. I is a function of the zenith angle of the gaussian sphere. Thus, we have circles of constant distortion 
values on the gaussian sphere. 

Fig. 7 shows our regular-pattern gradient map. 6 The curves shown are contours of constant distortion value. 
This map always applies to the shape-from-rcgular pattern problem. This is analogous to a situation in the 
shape-from-shading problem where the surface is known to be a Lambertian reflector and the light source is 
known to be near the viewer. 


2.5 Viewer-Centered Coordinate System and Spatial Coordinate System 

There arc two kinds of spheres: the image sphere and the gaussian sphere. The image sphere is a spatial 

coordinate system with which the image is specified. The gaussian sphere can be thought of as a kind of ruler 

for measuring surface orientation relative to die north-south axis of the sphere. When a pattern is at the origin 
_ - * “ 

This gaussian sphere is different from the image sphere. 

6 

We project the gaussian sphere onto a plane using the stcreographic projection. The stercographic projection puts the sphere on 
a plane and projects points on the sphere to the plane with respect to the north pole. Precise explanations arc given in [4,5,10]. 






Fig. 7 The Regular-Pattern Gradient Map 

Each circle corresponds to a constant distortion value I. The gaussian sphere is projected onto a plane by the 
stereographic projection. 


of the image sphere, the two spheres are aligned. In other words, the viewer coordinate system on the gaussian 
sphere is same as the spatial coordinate system of tire image sphere. When a pattern exists at some other point 
on the image sphere, then one must rotate the gaussian sphere so that its north pole points in that direction (V 
in Fig. 8). 

Fig. 8 shows the relationship between tire viewer coordinate system and the spatial coordinate system. 
[X, Y, Z) is tire spatial coordinate system, (x, y, V) is the viewer coordinate system. The direction V denotes 
the direction of line of sight of the viewer, or the position of a regular pattern. 
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% 


V(a 


viewed direction) 


(surface no 


(a spatial 
coordinate system) 


* (a viewer coordinate system) 



Fig. 8 Viewer-Centered Coordinate system (A gaussian sphere) and Spatial Coordinate System (An image 
sphere) 
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Formulas from spherical trigonometry give the following relationships; 


cos 0 n = cos 0 U cos u) -f~ sin 0 V sin uj cos(/r — it — <f> v ), 


sin {<j) v — <j> n ) 


sin to sin(u — 7 r — <f) v ) 


sin& 


CQs((f) v (j ) n ) 


COS U) — COS 0 n cos 0 V 
sinOnSmO,, 


whcie to and/x arc the zenith and azimuth angles of the surface normal in the viewer coordinate system, 9 n and 
(f> n are the zenith and azimuth angles of the surface normal in the spatial coordinate system, and 0 V and 4> v are 

the zenith and azimuth angle of the view vector in the spatial coordinate system. Thus, using Eq. 3 we can 
convert to and /.i into 0 n and (f> n . 

2.6 Obtaining Surface Orientation from Multiple View 
(How to Use Regular-Pattern Gradient Map) 

Surface orientation can be determined uniquely, given enough constraints. Surface orientation has two degrees 
of freedom. All points on the gaussian sphere arc candidates before observation of a distortion value. The 
measurement 01 a distoition value, J, eliminates one degree of freedom, leaving a one dimensional set of 
possible surface orientations on the gaussian sphere. 7 More than two distortion values at the same point will 
uniquely determine the surface orientation there. This idea comes from the technique of the Photo-Metric 
Stcrco[3,4], In practice, however, this texture stereo requires three different vantage points. 


There are two ways in which we can get more than three distorted values at a surface point. If there is one 
regular pattern on a surface, we can observe that regular pattern from several different vantage points. For 


7 ,. . 


ft is possible in principle to determine surface orientation uniquely front the projections of the two axis vectors however we 

do not follow that approach in this paper favoring instead to explore the use of the technique of shading analysis for analyzing 
texture gradients. b 
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example, this case occurs when you pass a building. The pattern of the building changes according to your 
position. We can also solve the problem from a single vantage point, if the surface is planer and contains more 
than three regular patterns, we can solve die problem from a single vantage point. Because the effect of several 
different vantage can be obtained by looking at different part of the planer surface. This case often occurs 
when we looks at a wall of a building that contains many windows. In either case surface orientation can be 
determined uniquely. 

The locus of possible surface orientation corresponding to a distortion value obtained at a point is a circle on a 
gaussian sphere. The center of the circle is the direction of the line of the sight in the spatial coordinate system. 
The radius of the circle is obtained from the distortion value, I, Similarly, we will get another circle by using a 
different vantage point. The intersection of these circles is the solution. 


passible 

[ surface orientations 

radius is obtained 

from an observed distortion value 

and the regular-pattern gradient 

V t ! map, 

Vhe center point denotes 
A viewer direction. 


Fig, 9 Texture Stereo 

The center of the circle is tine direction of line of sight in the image sphere (a spatial coordinates). The radius 
is obtained from the distortion value. 
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Since the stereographic projection preserves circles, a circle on the image sphere projects to a circle on the 
stereographic plane. Thus, we can work with circles on the stereographic plane. Their centers are the projected 
viewing directions, and their radii can be calculated fionr distoition values, I. 

Fig. 10 shows photographs of a square piece of paper taken from three different vantage points. Distoition 
values luh 'and h were calculated for these using Eq. 1. Namely, I — where 1 and g are lengths of the 

projected axis vectors on the pictures and r is the angle between two edges. FOr example, / — 25mm ,g — 
37mm, and r = 131 degrees in the first photo, giving / = 0.35. Since we know the viewing diiection, we 
can draw the circle 1. In the same way, we can draw circles 2 and 3. They intersect to each other at the point 
(0.0,0.0) which gives us the surface orientation of flic .square in the spatial coordinate system. 
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: A CAMERA 


\ Fig. 10 A Real Example of Texture Stereo 
/ In case 1 f:25 g:37 um:L31 azimuth :63 zenith:66 1:0.35, 

In case 2 f:47 g:49 tau: 120 azimuth:45 zenith:55 1:0.43, 

In case 3 f:75 g:52 inu:86 azimuth:. 0 zenith :45 1:0.47. 

The third circle docs not go through the intersection of the first two, 
but one can sec that the accuracy is rather high given that real dala 
is being used. 
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3 CONSTRAINTS AND THE PROPAGATION OF CONSTRAINTS 
(TO GET SURFACE ORIENTATION FROM A SINGLE VIEW) 

Constraint propagation is an important technique for problems involving analog values. Originally, this 
propagation idea was proposed for problems in designing electrical circuits by Sussman[6], If a node in a 
diagram has a value, that value can effect surrounding nodes, reducing the range of possible values neighbor¬ 
ing points can assume. In other words, the value propagates through a network, and becomes a new constraint 
on the surrounding nodes. This propagation technique is more effective than the search technique, when the 

• • . t .. i ... ... . J r ^ i: Jv;.s > • 1. 1 Vi l. A »... • *. * ' . ' •• : . VT * i • * ' ' v * = ~ > 

solution space has an large number of states. 

Smoothness gives us a tool for propagating constraints. If die surface is smooth, neighboring points must have 
nearly the same orientation. The smoothness constraint can be made explicit as a relationship between one 
node and its surrounding nodes. , ru 

., ' r ■ J i * * • • ........ _ 

3.1 The Smoothness Constraints (Propagation Constraints) 

Surface smoothness requires that the surface orientation be continuous over die image sphere. Our smooth¬ 
ness requirement is equivalent to the requirement that die surface height function be class c-1 on the image 

■■ ‘ i 

sphere [5]: 

1) surface is continuous on an image sphere, (c-0 w.r.l height) 

2) surface orientation is continuous on an image sphere, (c-l w.r.l height) 

Fiq. (5) is an "average" constraint which makes the smoothness requirement explicit. The definition of 
continuity can be expressed in a more convenient form; 

A function Fi is continuous at (x 0 , y„) 
if given an e > 0, 

there exits a S > 0 such that when (x — xff + (y — y 0 ) 2 < 8, 
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If we take our mesh size S 0 smaller that die S, Eq. (4) guarantees that 
/orVe, 3S 

\F{[x 0 S 0) y 0 ) -f- 1'i{x 0 S 0 , j/ 0 ) -f- Fi(x 0 , y 0 -f- 5 0 ) -(- F t [x ()1 y 0 5 0 ) 4 Fi{x 0 , y 0 )| <T 4e. (5) 

Thus, die difference of the surrounding values and four times the central value approaches zero as the mesh is 
made smaller. We call these constraints "average" constraints. A more precise discussion will be found in [5]. 


3.2 Regular-Pattern Gradient Map(Static Constraints) 
and A Propagation Algorithm 

The smoothness constraint can be used to reduce die locus of possible orientations on die gaussian sphere 
when only a single distortion value is obtained at each point. Each distortion value corresponds to a circle on 
the regular-pattern gradient map. Points on die circle arc possible surface orientations at the point. We can 
make a propagation algorithm from the smoothness constraints to further reduce this locus of possible surface 
orientations. 
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Let 


F ?r = k, " ■+ (n ». ■Gy-i f - , -,c-.), 

*»i 

G <r - G h - +(jjx^ - n , G i s .), 


where 


F* 


^ +1 , i +F?_ 1)i H-n i+1 +F^._ 1 


r * n_ G i+i,j + G L-U + G ”i +1 + G "i -1 
*.i 4 

F and G denote suiface orientations, Ii t j is a distortion value at the point (i,j) obtained from observation. R 
is the regular-pattern gradient map. Eq. ( 6 ) has three kinds of input; kj (distortion value) comes from local 
observation, and the other two ( F* and G* )are propagated from surrounding nodes. In other words, this 
propagation algorithm makes a new estimate of the center node based on prior estimates of surrounding nodes 
and the observed distortion value( See Fig. 11) A more precise discussion on Eq. ( 6 ) is found in [5]. 


old values 

at the surrounding 

points 


the 

propagat Lon 
algorithm 


a d ist o rtio n - 

value: 

at the center 
point 


a new value 
at the center 
point 


Fig. 11 A Propagation Algorithm 

This algorithm is used iteratively. After the state of the estimation readies equilibrium, the algorithm terminates. 
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3.3 Occluding Boundary Constraints (External Constraints) 

At an occluding boundary we can determine the surface normal uniqucly[5]. The line of sight is tangent to 
the object at the occluding boundary. So the surface normal is perpendicular to the line of sight. Since the 
line of sight of the viewer also lies at the tangent plane, die tangent plane is projected as a tangent line of the 
contour on the image plane. From the orientation of the occluding boundary we can determine surface normal 
uniquely at the occluding boundary 8 

This occluding information gives us the strong boundary conditions We need to start die propagation algo¬ 
rithm. The propagation constraints only convey information. In other words, the propagation constraints 
determine a relative relationship between nodes; but they cannot create new information. The occluding 
boundaries fix the surface orientation at those points. From these boundaries we can propagate the informa¬ 
tion inward over die image points adjusting the propagated value with the local constraints provided by the 
distortion value at each point. 

This propagation method is more suitable than Horn’s method[8] in the shape-from-regular-pattern problem. 
While it is possible to use Horn’s method which starts at a singular point, and proceeds along characteristic 
lines, his mediod is not able to use the occluding information, which is available in this problem. Thus, errors 
accumulate to much greater extent and the quality of the solution depends rapidly over long stretches. On 
die other hand, the method presented here utilizes constraints at both end point. 9 The use of the occluding 
boundary constraint makes our method is less sensitive to noise than Horn’s. In addition the shape-from- 
regular-pattern problem provides fewer local constraints than the shape-from-shading problem. This shortage 
increases the need to utilize all available constraints. 


o 

Note that [5] used the orthographic projection, while wc use the spherical projection here. These projections are different from 
each other. However, the sphere projection has a image plane which is perpendicular to the lines of sight of die viewer everywhere. 
Thus, the same discussion is applicable to the occluding boundary. 

^Actually, there are no start points and end points in our method. All information from occluding boundaries and a singular 
point is provided to the algorithm, theoretically, however, a piece of information at an occluding boundary propagates inwards 
and interacts with information from the other occluding boundaries. 
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EXPERIMENTS 


4.1 Experiment 1 (A Infinite Planar Surface) 

Consider a plane covered by many squares. The squares are projected onto an image sphere using the spheri¬ 
cal projection. Fig. 12 shows the synthesized image. For graphical clarity, the image sphere is mapped onto 
a planar surface using the stereographic projection 10 This maps an infinite plane to tire interior of the unit 
circle 11 . Note that the infinite horizon line maps to a circle. This circle is equivalent to an occluding boundary. 
Thus, the surface orientation is fixed on that circle. 
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Fig. 12 The synthesized image 
A plane is covered by many squares 

j J 

j.Thc squares are pd 
! an image sphere* 


J Note that there are two spheres; the image sphere and the gaussian sphere. Both of them are projected onto a plane using 
the stereographic projection. 

**’This figure resembles a picture taken using a fish-cyc lens, 
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The process contains three main stages (Fig. 13). The first stage obtains distortion values from a picture and 
sutface oiientations on the occluding boundaries. The second stage is tire main one. In this stage we compute 
surface orientation for all points using tire propagation method. Since the surface orientations obtained from 
the second stage are expressed in a viewer-centered coordinate system, a third stage is required to convert the 
result to the spatial coordinate system. 



Fig. 13 Three stages 

® obiain distortion values and surface orientations on the occluding boundary 

(2) lo compute surface orientation for all points. w ** 

(3) To convert a viewei coordinate system to the spatial coordinate system. 
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We get the distortion using 


/osin r 


where / and g are lengths of the projected squares edges and r is the angle between the two edges. Sec 
3.4 discusses die way in which surface orientations at the occluding boundaries is obtained. These distortion 
values and surface orientations are input to the propagation algorithm. Fig. 14 shows the distortion values and 
Fig. 15 shows surface orientations from occluding boundaries. 
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Fig. 14 The obtained distortion values 


Fig. 15 Surface orientations from occluding boundaries. 


The propagation algorithm estimates surface orientation at each point and is applied iteratively, at each node 
referring to the values of surrounding node and die corresponding distortion value. These estimates are the 
input information to the next iteration. When the entire system of nodes reaches a stable state, the algorithm 















































Ikeuchi 


26 


SHAPE FROM REGULAR PATTERNS 


stops. 

There are two parts to the algorithm, a propagation part, and a regular-pattern gradient part. The propagation 
part conveys information from surrounding nodes to the center node. The regular-pattern gradient part uses 
the gradient map and a distortion value to determine the locus of allowed orientation. The mathematical 
details (convergence, effect of open boundary etc.) will be found in [5]. 

Fig. 16 shows the needle diagrams obtained for a planar surface. They arc expressed using the viewer coor¬ 
dinate system. The direction of the viewer’s line of sight is taken as the z-axis, although the physical direction 
is different at each image point, because we are using the sphere projection. Fig. 17 is the planar surface 
depicted in the viewer coordinate system. 
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Fig. 16 The result needle diagram 


Fig. 17 The planar surface depicted 
in the viewer coordinate system. 
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4.2 Experiment 2 (A Golf Bali) 

We will compute the surface orientation for a golf ball using its surface texture. A golf ball has many small 
circles on its surface. The distortion of these circles can be used to recover local surface orientations. Fig. 19 
shows input the picture. 



Fig. 19 A golf ball 

The distortion of these circles on the golf bail can be used to recover local surface orientations. 
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We can get distortion values from this picture by drawing two parallel lines in an arbitrary orientation tangent 
to each circle. We connect the resulting tangent points on each circle and use this connecting line as one of the 
axis vectors. We then draw a line parallel to tire tangential lines through the center of the first axis vector. The 
part of this line within the projected circle is the second axis vectors. We can compute tire distortion values 
using Eq. 1. Fig. 21 shows the distortion values obtained. 



Fig. 20 How to get a pair of axis vectors from a circle. 



Fig. 21 Obtained distortion values 
bach needle corresponds to a little 
there. 


circle on the ball and expresses how much the little 


circle is distorted 
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We can calculate surface orientation by using the algorithm. In this case the object is very small compared with 
the distance between the viewer and the object. Thus, there is of no need to convert the result in the viewer 
coordinate system to the spatial coordinate system. Fig. 22 shows the resulting surface. 
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Fig. 22 A computed results 
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5 CONCLUDING REMARKS 

(1) Wc proposed a sphere projection to treat tire perspective view. 

(2) We defined a distortion value for a regular pattern and proposed a regular-pattern gradient map using 
this measure. 

(3) We showed that die propagation algorithm can successfully determine surface orientations from regular- 
pattern gradients using the sphere projection, the regular-pattern gradient map, and occluding boundary 
constraints. 


The propagation of constraints is a powerful idea in vision. Each node always has some relationship with 
the neighbors. Clarification of this relationship often gives us a key to the solution of vision problems. The 
propagation of constraints from neighboring nodes cuts down die area of possible solutions at die node. In 


vision the number of nodes is very large and each node takes analog values. This means that die solution soace 
has a large number of states. Thus, the propagation technique is more effective than the search technique. 
Even if the propagation technique cannot determine a unique solution, it can reduce the number of possible 
solutions so that a subsequent search technique can be used more effectively. 


There are several examples in vision where this technique has been used successfully. Marr and Poggio’s first 
stereo algorithm used this idea [9], Their continuity rule is an example of a positive propagation requiring diat 
neighboring points have similar values (disparity values). Their uniqueness rule is an example of a negative 
propagation; A value at a node inhibits for another nodes from taking the same value. These two rules 
describe the relationship between nodes. Ikeuchi’s shape from shading algoridim[5] has die same form as the 
shape from regular-patterns algorithm presented here. Third, Horn’s lightness reconstruction algorithm [11] 


also used a propagation technique. The G operator calculated a lightness value at the center node based on 
both old lightness values at neighboring points and output from a laplacian operator at die center node. 


The propagation of constraints technique is suited for array processing. The relationship between nodes 
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is homogeneous over an array. We can connect each node with the surrounding nodes in die same way 
everywhere, without regarding die position of nodes. An essential operation of the above mendoned algorithm 
is to compute an average of neighboring points. Thus, parallel hardware for array addition can implement the 
above mentioned algorithms very easily. The result can be obtained very rapidly. 
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Fig. A.l shows the relationship between the viewer and a regular pattern; the viewer is at V — (0, 0, cos 0 V ) 
and the texture element is at R — (—cos<f> v smO v , — sm<j> v sm0 v , 0) 11 .Thus, the viewer vector V is 

V — (cos <j> v sin 0 V , sin cf> v sin 0 V , cos 0 V ). 

The surface normal of a plane on which the regular patter lies can be denoted as 

N — (cos <f> sin 0, sin <f> sin 0, cos 0) 

: - * . i , • r • r • v • 

in the spatial coordinate system. We can take two base vectors of this surface as 


4 

X = (sin <f>, — cos 4>, 0), 

4 . 

Y — (cos <j> cos 0, sin <f> cos 0, — sin 0). 



*4 

X is a unit vector which lies on the intersection between die image plane and the surface on which the regular 
pattern lies. Thus, a general form for two axis vectors of the regular pattern is A,B where; 


4 4 -* 

A — (X cos q -f- Y sino), 

B — (— X sina -(- Y cos o), 



where a denotes the orientation of the texture element in the surface. Variation of this a causes changes like 
those shown in Fig. 6. 


4 4 4 

Projections a and b of axis vectors, A and B onto the image plane are 


a — k(A — (A ■ V)V), 
b == k(B — (B ■ $)V), 



where k is a scale factor, which depends on the distance between die regular patter and the viewer. 


We want to find an intrinsic value dependent on the surface normal but independent of die rotation angle a 
and the scaling factor k. With this intrinsic value we will be able to construct a regular-pattern gradient map. 


n. 


Hie following calculation will be easier if the z-axis is the same as the direction of the viewer. 
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Consider the ratio of the cross product of the two projected axis vectors over the sum of the square of the 
lengths of the two projections. 

The cross product of die axis vector’s projections onto die image plane is: 

|a X b\ = k 2 \{V X X)(Y • V) - (V X Y)(X • V)\, 

= lc 2 \V ■ {X X Y)\, (A.4) 

— k 2 cosw, 

where to denotes the angle between the direction of the line of sight of the viewer and die direction of the 
surface normal. Thus, the cross product of the projected axis vectors does not depend on the rotation angle 
a, and is proportional to die cosine of the angle between the direction of line of sight of the viewer and the 
direction of the surface normal. 


The sum of the square of the lengths of the projected vectors is: 

|a| 2 + |6| 2 = k\2 — [\X • V\ 2 + \Y * V\ 2 ]), 

— (cos 0 V cos 0 4" sin^sin^cos^ ~~ ^)) 2 ), ("4*5) 

= k 2 {\ + cos 2 a;} # 


where we used a cosine formula from spherical trigonometry. 


We can cancel die scaling factor by dividing Eq.(A.4) by F.q.(A.5) obtaining the distortion measure /: 


a X 6| 


costa 


ap + |S | 2 i-)- cos 2 u/ 


(A.6) 


I 
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A.2 Conversion from the Perspective Projection to the Sphere Projection 

Wc can take two base vectors on the surface of an object. Every vector on the surface can be expressed using 
these two vectors. Let one vector be e L as on the extended plane of the meridian circle and let the other vector 
h as perpendicular to die first vector t\. 

The second vector e 2 is projected in the same way in both the perspective projection and the sphere projection. 
The first vector is projected as in Fig. A.2. 



Paphere — |ei|sinW, 

D _ jeiI sinw (A.7) 

* perspective — ^ > 

COS u v 

where P ap here and P pe rapective are the projected lengths of q onto die image plane under the sphere projec¬ 
tion and the perspective projection, respectively, w denotes the angle between the direction of Viewer’s line of 
sight and the direction of the surface normal. 0 V is the zenith angle of the image point based on the direcdon of 
die optical axis of die image plane. 

Thus, 
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* sphere ^perspective COS 


(A.8) 


Eq. (A.8)expresses the relationship between the sphere projection and the perspective projection. 


Eq. (A.7) also explains why tire perspective projection causes the problem discussed in Chapter 2. Imagine an 
infinite plane perpendicular to the optical axis of the perspective projection. The infinite plane contains many 
squares. Under the perspective projection these squares always projected as squares. Which is contrary to our 
experience. We actually see squares as rectangles whose shape depends on their positions. In this case to is 
always as same as (f + 0 V ). Thus, from Eq. (A.7) we get 


perspective 



Thus, the perspective projection does not exhibit foreshortening effects. 





